Derivative free optimization method

Derivative free optimization (DFO) methods are typically designed to solve optimization problems whose objective function is computed by a “black box”; hence, the gradient computation is unavailable. Each call to the “black box” is often expensive, so estimating derivatives by finite differences may be prohibitively costly. Finally, the objective function value may be computed with some noise, and the finite differences estimates may not be accurate. All above properties, such as relatively expensive “black box” computations and presence of noise, are characteristic of the Cycle-Tempo optimization problems. However, it is the noise which creates most difficulty in applying gradient based methods to these problems. The derivative free optimization method which we use approximates the objective function explicitly without approximating its derivatives. The theoretical analysis presented below assumes that no noise is present. However, extensive experiments and intuition support the claim that robustness of DFO does not suffer from presence of moderate level of noise. Various other methods were developed recently to handle similar classes of problems (see [11], [12], [8], [6], [1]). In [1] a modification of a quasi-Newton method that accommodates noise in the objective function is considered. However, the analysis in [1] assumes that the level of noise reduces to zero when optimal solution is approached. This assumption is not realistic in case of Cycle-Tempo. The DFO method that we use here compares favorably with many other existing derivative free methods (both in speed and in accuracy). It is described in [2] and [3] and is essentially a combination of a trust region framework with quadratic interpolation of the objective function. Using polynomial interpolation models within trust region frameworks has been proposed earlier in [8], [6] and [5]. The main distinction of the method in [2] and [3] from these earlier methods is in the approach to handling the geometry of the interpolation set. In Subsection 3 we briefly describe the essence of this approach; for more detailed analysis the reader is referred to [2]. First, we describe the trust region framework.